Imaging through multimode fiber (MMF) provides high-resolution imaging through a fiber with cross section down to tens of micrometers. It requires interferometry to measure the full transmission matrix (TM), leading to the drawbacks of complicated experimental setup and phase instability. Reference-less TM retrieval is a promising robust solution that avoids interferometry, since it recovers the TM from intensity-only measurements. However, the long computational time and failure of 3D focusing still limit its application in MMF imaging. We propose an efficient reference-less TM retrieval method by developing a nonlinear optimization algorithm based on fast Fourier transform (FFT). Furthermore, we develop an algorithm to correct the phase offset error of retrieved TM using defocused intensity images and hence achieve 3D focusing. The proposed method is validated by both simulations and experiments. The FFT-based TM retrieval algorithm achieves orders of magnitude of speedup in computational time and recovers 2286 × 8192 TM of a 0.22 NA and 50 μm diameter MMF with 112.9 s by a computer of 32 CPU cores. With the advantages of efficiency and correction of phase offset, our method paves the way for the application of reference-less TM retrieval in not only MMF imaging but also broader applications requiring TM calibration.